Deriving Receptive Fields Using an Optimal Encoding Criterion
نویسنده
چکیده
An information-theoretic optimization principle ('infomax') has previously been used for unsupervised learning of statistical regularities in an input ensemble. The principle states that the inputoutput mapping implemented by a processing stage should be chosen so as to maximize the average mutual information between input and output patterns, subject to constraints and in the presence of processing noise. In the present work I show how infomax, when applied to a class of nonlinear input-output mappings, can under certain conditions generate optimal filters that have additional useful properties: (1) Output activity (for each input pattern) tends to be concentrated among a relatively small number of nodes. (2) The filters are sensitive to higher-order statistical structure (beyond pairwise correlations). If the input features are localized, the filters' receptive fields tend to be localized as well. (3) Multiresolution sets of filters with subsampling at low spatial frequencies related to pyramid coding and wavelet representations emerge as favored solutions for certain types of input ensembles.
منابع مشابه
An Efficient Bayesian Optimal Design for Logistic Model
Consider a Bayesian optimal design with many support points which poses the problem of collecting data with a few number of observations at each design point. Under such a scenario the asymptotic property of using Fisher information matrix for approximating the covariance matrix of posterior ML estimators might be doubtful. We suggest to use Bhattcharyya matrix in deriving the information matri...
متن کاملAre V1 Simple Cells Optimized for Visual Occlusions? A Comparative Study
Simple cells in primary visual cortex were famously found to respond to low-level image components such as edges. Sparse coding and independent component analysis (ICA) emerged as the standard computational models for simple cell coding because they linked their receptive fields to the statistics of visual stimuli. However, a salient feature of image statistics, occlusions of image components, ...
متن کاملMultidimensional Encoding Strategy of Spiking Neurons
Neural responses in sensory systems are typically triggered by a multitude of stimulus features. Using information theory, we study the encoding accuracy of a population of stochastically spiking neurons characterized by different tuning widths for the different features. The optimal encoding strategy for representing one feature most accurately consists of narrow tuning in the dimension to be ...
متن کاملScale-Space Theory for Auditory Signals
We show how the axiomatic structure of scale-space theory can be applied to the auditory domain and be used for deriving idealized models of auditory receptive fields via scale-space principles. For defining a time-frequency transformation of a purely temporal signal, it is shown that the scale-space framework allows for a new way of deriving the Gabor and Gammatone filters as well as a novel f...
متن کاملWhat Are the Invariant Occlusive Components of Image Patches? A Probabilistic Generative Approach
We study optimal image encoding based on a generative approach with non-linear feature combinations and explicit position encoding. By far most approaches to unsupervised learning of visual features, such as sparse coding or ICA, account for translations by representing the same features at different positions. Some earlier models used a separate encoding of features and their positions to faci...
متن کامل